Grouped-Query Attention

KV Cache reduces duplicated computation in autoregressive LLM inference by storing previously computed Key and Value tensors, but creates a memory bottleneck as context length grows. To address this, …

Google's Gemma 2 models demonstrate that architectural efficiency can deliver competitive performance with fewer parameters. The 27B model rivals models twice its size through hybrid attention, Groupe…

JetBrains released Mellum2, a 12-billion-parameter Mixture-of-Experts model with 2.5 billion active parameters per token, under the Apache 2.0 license. The model is specialized for software engineerin…